Relevant Expressions in Large Corpora

The automatic extraction of statistically relevant expressions of any language from raw (non-annotated) corpora is a very useful task, specially when it may be used for the study of data from old texts, given the unavailability of informants, the large amount of graphic variants found in that kind of texts and the scarcity of annotated texts. Furthermore, statistically based extraction of complex lexical units and relevant expressions from non-annotated corpora proves to be more precise and less expensive than usual methods that require the search of syntactic patterns from annotated corpus. Moreover, it is also interesting for linguistic, historical, cultural and literary studies. From the computational point of view, the extraction of those expressions enables the necessary means for incorporating complex lexical units in specialised lexica, which is also very useful for the development of the Natural Language Processing (NLP). This paper presents a study of the relevant expressions taken from a Medieval Portuguese corpus using a statistical method based on three main tools: The LocalMaxs algorithm, a statistical measure and the Fair Dispersion Point Normalisation concept. In order to assess our results we have tested several statistical measures, namely Specific Mutual Information (SI) (Church K.; K. Hanks (1990)), SCP (Silva J.F.; G.P. Lopes (1999)), Dice coefficient (Dice L. (1945)), Loglike coefficient (Dunning T. (1993)), and  coefficient (Gale W.A.; K.W. Church (1991)). We have compared the statistically relevant expressions extracted by these measures and evaluated the results and their relevance both for NLP and other domains of research, namely Linguistics, History, Cultural and Literary Studies. It should be stressed that the LocalMaxs algorithm and the Fair Dispersion Point Normalisation enabled approximately 20% improvement of precision obtained, except for the Dice and Loglike measures.